Start nspawn workers after reboot on RHEL and recover stopped workers by Copilot · Pull Request #437 · Azure/AKSFlexNode

Copilot · 2026-11-10T18:10:51Z

Summary

RHEL presets leave machines.target disabled, preventing workers from starting after reboot. The daemon still reported success because it checked persisted settings, not worker liveness.

Boot startup: Add Wants=machines.target to host and container daemon units.
Preflight: Warn on disabled targets; fail on missing or masked targets.
Worker health: Verify the active worker before reporting success, restart it without repaving if stopped, and return reconcile errors when inspection or recovery fails.

Validation

Full test suite, affected-package race tests, vet, documentation checks, build, and CLI smoke test passed. CodeQL and secret scans were clean.

golangci-lint was unavailable; live RHEL reboot testing remains outstanding.

Changelog

I added a user/operator-facing entry under Unreleased in CHANGELOG.md, or this change does not require a changelog entry.

Fixes [Bug] machines.target disabled on RHEL hosts #436

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot

🟡 Changes recommended

Worker failures can go undetected and unrecovered, while the new preflight check blocks online bootstrap before required packages can be installed.

2 open issues

What changed in this PR

This PR addresses the reboot-startup portion of #436 by making the Flex Node daemon request machines.target, allowing enabled nspawn workers to start even when RHEL presets disable the target.

Changes:

Add machines.target dependencies to host and container daemon units.
Register a preflight check for target availability and enablement.
Add checker tests, update service snapshots, and document preflight behavior.

File	Description
`pkg/daemon/testdata/aks-flex-node-agent.service.golden`	Update the expected host-unit dependency.
`pkg/daemon/testdata/aks-flex-node-agent-arc.service.golden`	Update the expected Arc-enabled unit dependency.
`pkg/daemon/preflight.go`	Add target-state validation.
`pkg/daemon/preflight_test.go`	Test target states and daemon-unit dependencies.
`pkg/daemon/assets/aks-flex-node-agent.service`	Request worker startup through `machines.target`.
`pkg/daemon/assets/aks-flex-node-agent-container.service`	Add the same dependency for container deployments.
`pkg/cmd/preflight/preflight.go`	Register the new preflight check.
`docs/usage/operations.md`	Explain target-check errors and warnings.

_{Review effort: Balanced}

Philip Lombardi (plombardi89)

Thanks for this. The boot fix itself works. Requesting changes for the preflight check and some housekeeping.

Blocking

The machines-target check fails preflight on a fresh host where systemd-container isn't installed yet, although bootstrap installs it and unbounded's own checks only warn about it. Inline on pkg/daemon/preflight.go:37, with a reproducer on RHEL 9.8.
Housekeeping before merge:
- The description is out of date. It still lists the "Worker health" changes that 695b06b removed.
- Fixes #436 will close the issue, but only half of it is fixed. The issue's second request (the agent noticing a stopped worker instead of reporting success for three days) was dropped. Please use Refs #436 and open a follow-up issue, or note on #436 that this part won't be done.
- No CHANGELOG.md entry. There's an [Unreleased] section, and this change is user-facing: a fix (workers now start after reboot on RHEL) and a new preflight check.

Verified: Wants=machines.target. I tested it on systemd 252 (Rocky Linux 9.8, the same systemd as the RHEL 9.8 hosts in #436):

The worker was the real systemd-nspawn@.service, with only its process replaced by sleep.
The agent unit was the actual golden unit from main or from this PR, again with only ExecStart replaced.
machines.target was disabled and systemd-nspawn@kube1.service was enabled, as on the reporter's hosts.
Each scenario was a reboot.

Scenario	`machines.target`	Worker	Agent
Agent unit from `main`	inactive	never started (reproduces #436)	active
Agent unit from this PR	active	active	active, without waiting for the worker
PR's container-variant unit, `containerd` start fails	active	active	not started (dependency failed)
Worker disabled and stopped on purpose, as unbounded does when it stops a machine	active	stays stopped	active

The upgrade path also heals hosts that are already stuck. Writing the new unit and running daemon-reload alone starts nothing, but the agent restart that follows an upgrade starts machines.target and the stopped worker. Stopping the agent doesn't stop either of them.

After=machines.target: no change needed. Unbounded's own daemon unit has After= as well as Wants=. Adding it here would hold the agent until the worker's start finishes; I tested that, and a worker whose start fails doesn't block the agent. A start that hangs would hold it for up to the worker's start timeout, 90 seconds by default for the Type=notify nspawn unit. I found nothing in pkg/daemon that assumes the worker is up at startup, and the worker's pre-start/post-start hooks don't depend on the agent unit. So Wants= alone is the better choice here.

The remaining gap is a real reboot of a real RHEL host with a real nspawn worker, which the PR description already lists as outstanding.

hbc (bcho) · 2026-11-10T20:34:31Z

Copilot address pr comments

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot

🟡 Changes recommended

Preflight incorrectly rejects valid linked systemd targets, and runtime-only links also need a persistence warning.

2 open issues

1 resolved since last review

Reconciles succeed without checking or recovering the worker

_{Review effort: Balanced}

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Human assessment is needed to weigh the operational impact of limited preflight failure diagnostics.

1 open issue

Accept linked systemd units during preflight

1 resolved since last review

Fail preflight when containerized bootstrap lacks the host target

_{Review effort: Balanced}

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Human assessment is needed to weigh an unpublished low-severity concern about diagnosing systemd inspection failures.

No open issues 🎉

1 resolved since last review

Accept linked systemd units during preflight

_{Review effort: Balanced}

hbc (bcho) · 2026-11-11T21:25:37Z

Copilot resolve conflicts

…target-issue # Conflicts: # CHANGELOG.md Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot

🟡 Changes recommended

The preflight result omits the underlying error, leaving operators without the cause when systemctl cannot be started.

1 open issue

Include the underlying error in preflight results · New

_{Review effort: Balanced}

Copilot · 2026-11-11T21:31:29Z

Copilot resolve conflicts

Resolved the changelog conflict by merging main in f05131a. Targeted Go tests and documentation checks passed.

Philip Lombardi (plombardi89)

Approving. The preflight fix is verified (details in the resolved thread), and the boot fix holds up on a real RHEL host (below).

Before merging, please update the PR's title and description. They still describe work that 695b06b removed:

Title: drop "and recover stopped workers", for example "Start nspawn workers after reboot on RHEL".
Body:
- remove the "Worker health" bullet
- update the "Preflight" bullet to the online/offline/containerized behavior
- replace "live RHEL reboot testing remains outstanding" (results below)
- check the changelog box
Use Refs #436 instead of Fixes #436, and open a follow-up issue for the second request (the agent noticing a stopped worker). The test below reproduced it: with the worker down and the node NotReady, the agent kept logging decision=ReportSucceeded reason="goal state is applied". Merging with Fixes would close the issue with that still open.

Real RHEL host test. Setup:

an Azure VM from RedHat:RHEL:9_8-gen2, joined to a new AKS cluster with the repo's e2e token-join scripts
the boot fix from this PR (the unit files are unchanged since 69f1cc6)
machines.target disabled by the RHEL preset, with systemd-nspawn@kube1 enabled under it, as in #436

Each row is a reboot unless noted:

Test	`machines.target`	Worker	Node
This PR's unit	active (still `disabled`)	started; its kubelet was running 13 s after boot	Ready
Pre-PR `Wants=` line restored	inactive	never started	`NotReady`; the agent kept reporting `ReportSucceeded`
Agent restart, as after an upgrade, with the worker down (no reboot)	started	started	Ready again
The unit as the agent rewrote it at startup	active	started	Ready

Also from that host:

Wants= alone is enough. The worker was active 3 s before the agent even without After=machines.target, because the agent also waits for network-online.target.
The project's own e2e token-join flow hits the old preflight error on RHEL. It installs host packages only with apt-get, so it reached preflight without systemd-container and failed on machines-target. With the current head that's a warning.

Separate from this PR: the agent crash-loops on RHEL with SELinux enforcing. Enforcing is the default on Azure's RHEL images, so I set the test VM to permissive to get through the test.

The error: initialize agent binary layout: initialize daemon compatibility symlink: symlink /opt/unbounded/agent/lib/aks-flex-node/aks-flex-node-current /opt/unbounded/agent/bin/.aks-flex-node…/tmp.symlink: permission denied
A/B test: enforcing fails, permissive works.
Context: the agent runs as init_t, and its binary under /opt/unbounded/agent/lib is labeled lib_t. No AVC is logged, even with dontaudit rules disabled.
Suggestion: this probably deserves its own issue.

Initial plan

72a1624

Copilot AI assigned Copilot and hbc (bcho) Nov 10, 2026

Copilot started work on behalf of hbc (bcho) November 10, 2026 18:11

View session

Copilot AI linked an issue Nov 10, 2026 that may be closed by this pull request

[Bug] machines.target disabled on RHEL hosts #436

Open

Start machines target with daemon and recover stopped workers

22bd80d

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix machines.target disabled issue on RHEL hosts~~ Start nspawn workers after reboot on RHEL and recover stopped workers Nov 10, 2026

Copilot finished work on behalf of hbc (bcho) November 10, 2026 18:20

Copilot AI requested a review from hbc (bcho) November 10, 2026 18:20

hbc (bcho) reviewed Nov 10, 2026

View reviewed changes

Comment thread pkg/daemon/repave_reconciler.go Resolved

Copilot started work on behalf of hbc (bcho) November 10, 2026 18:24

View session

Remove worker liveness recovery per review

695b06b

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot finished work on behalf of hbc (bcho) November 10, 2026 18:31

Copilot AI requested a review from hbc (bcho) November 10, 2026 18:31

hbc (bcho) reviewed Nov 10, 2026

View reviewed changes

Comment thread pkg/daemon/preflight.go Resolved

Copilot started work on behalf of hbc (bcho) November 10, 2026 19:13

View session

Copilot finished work on behalf of hbc (bcho) November 10, 2026 19:14

Copilot AI requested a review from hbc (bcho) November 10, 2026 19:14

Copilot started work on behalf of hbc (bcho) November 10, 2026 19:15

View session

Note planned move of machines target check to Unbounded

69f1cc6

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot finished work on behalf of hbc (bcho) November 10, 2026 19:17

Copilot AI deployed to e2e-testing November 10, 2026 19:31 Active

Philip Lombardi (plombardi89) marked this pull request as ready for review November 10, 2026 20:18

Copilot AI balanced review requested due to automatic review settings November 10, 2026 20:18

Copilot started reviewing on behalf of Philip Lombardi (plombardi89) November 10, 2026 20:19

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread pkg/daemon/assets/aks-flex-node-agent.service Resolved

Comment thread pkg/daemon/preflight.go Resolved

Philip Lombardi (plombardi89) requested changes Nov 10, 2026

View reviewed changes

Comment thread pkg/daemon/preflight.go Resolved

Copilot started work on behalf of hbc (bcho) November 10, 2026 20:35

View session

Handle missing machines target during online preflight

85ea2de

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot started reviewing on behalf of hbc (bcho) November 11, 2026 19:16

View session

Copilot AI reviewed Nov 11, 2026

View reviewed changes

Comment thread pkg/daemon/preflight.go Resolved

Copilot started work on behalf of hbc (bcho) November 11, 2026 19:20

View session

Copilot finished work on behalf of hbc (bcho) November 11, 2026 19:21

Copilot AI requested a review from hbc (bcho) November 11, 2026 19:21

hbc (bcho) requested changes Nov 11, 2026

View reviewed changes

Comment thread CHANGELOG.md Resolved

Copilot started work on behalf of hbc (bcho) November 11, 2026 20:58

View session

Move machines-target changelog entry to Fixed

fce7d11

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 11, 2026 21:02

Copilot started reviewing on behalf of hbc (bcho) November 11, 2026 21:02

View session

Copilot AI reviewed Nov 11, 2026

View reviewed changes

Copilot finished work on behalf of hbc (bcho) November 11, 2026 21:06

Copilot AI requested a review from hbc (bcho) November 11, 2026 21:06

Copilot started work on behalf of hbc (bcho) November 11, 2026 21:06

View session

Warn on unfamiliar machines target states

416d340

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 11, 2026 21:13

Copilot started reviewing on behalf of hbc (bcho) November 11, 2026 21:13

View session

Copilot finished work on behalf of hbc (bcho) November 11, 2026 21:16

Copilot AI reviewed Nov 11, 2026

View reviewed changes

Copilot started work on behalf of hbc (bcho) November 11, 2026 21:26

View session

Merge remote-tracking branch 'origin/main' into copilot/fix-machines-…

f05131a

…target-issue # Conflicts: # CHANGELOG.md Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 11, 2026 21:27

Copilot started reviewing on behalf of hbc (bcho) November 11, 2026 21:27

View session

Copilot AI reviewed Nov 11, 2026

View reviewed changes

Comment thread pkg/daemon/preflight.go Resolved

Copilot finished work on behalf of hbc (bcho) November 11, 2026 21:31

Copilot AI requested a deployment to e2e-testing November 11, 2026 21:35 In progress

Philip Lombardi (plombardi89) approved these changes Nov 11, 2026

View reviewed changes

hbc (bcho) approved these changes Nov 11, 2026

View reviewed changes

Conversation

Copilot AI commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Changelog

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Philip Lombardi (plombardi89) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hbc (bcho) commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

hbc (bcho) commented Nov 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI commented Nov 11, 2026

Uh oh!

Philip Lombardi (plombardi89) left a comment

Choose a reason for hiding this comment

Uh oh!

This branch is being deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Nov 10, 2026 •

edited

Loading